Survey on Drowsiness Detection and Alert System using ML Approach

Abstract

Many of the accidents occur due to drowsiness of drivers. It is one of the critical causes of roadways accidents now-a-days. Latest statistics say that many of the accidents were caused because of drowsiness of drivers. Vehicle accidents due to drowsiness in drivers are causing death to thousands of lives. More than 30% of accidents occur due to drowsiness. For the prevention of this, a system is required which detects drowsiness and alerts the driver which saves the life. In this project, we present a scheme for driver drowsiness detection based on visual information and Machine Learning. In this, the driver is continuously monitored through a webcam. This system is used to locate, track, and analyse both the drivers face and eyes, a scientifically supported measure of drowsiness associated with slow eye closure. The model extracts the driver\'s face and predicts the blinking of the eye from the eye region. If the blinking rate is high then the system alerts the driver with a sound.

Introduction

I. INTRODUCTION

A. Motivation

Increasing accidents due to unconsciousness or due to a driver’s diminished vigilance is a serious contribution to overall accidents in the world. However major accidents in the world are related to driver fatigue or drowsiness. Car accidents associated with driver fatigue are more likely to be serious, leading to serious injuries and deaths.  It is estimated that 40% of all traffic accidents have been caused by drowsiness. It was demonstrated that driving performance deteriorates with increased drowsiness with resulting crashes constituting more than 20% of all vehicle accidents. The performance of the driver also deteriorates with drowsiness.

Every fraction of seconds drowsiness can turn into dangerous and life- threatening accidents that may lead to death also. To prevent this type of incidents, it is required to monitor driver’s alertness continuously and when it detects drowsiness, the driver should be alerted. Through this we can reduce the significant number of accidents and can save the lives of people.

B. Problem Definition

To detect the drowsy condition of the driver using ML approach and alert the driver.

II. LITERATURE SURVEY

There are different approaches to identify the drowsiness state of the driver. They can be categorized into the following three main categories:

1. Behavioural parameters-based techniques: Measuring the driver’s fatigue without using non-invasive instruments comes under this category. Analysing the behaviour of the driver based on his/her eye closure ratio, blink frequency, yawning, position of the head and facial expressions. The current parameter used in this system is the eye-closure ratio of the driver.
2. Vehicular parameters-based techniques: Measuring the fatigue nature of the driver through vehicle driving patterns comes under this category. These parameters include lane changing patterns, steering wheel angle, steering wheel grip force, vehicle speed variability and many more.
3. Physiological parameters-based techniques: Measuring the drowsiness of the driver based on the physical conditions of the driver falls under this category. Such parameters may be respiration rate, heart-beat rate, body temperature and many more.

In this section, we have discussed various methodologies that have been proposed by researchers for drowsiness detection and blink detection during the recent years.

A. Drowsiness and Fatigue  

Drowsiness is where a person is in the middle of an awake and sleepy state. This situation leads the driver to not giving full attention to their driving. Therefore, the vehicle can no longer be controlled due to the driver being in a semi-conscious state.  According to research mental fatigue is a factor of drowsiness and it causes the person who experiences drowsiness to not be able to perform because it decreases the efficiency of the  brain to respond towards sudden events.

B. Electroencephalography (EEG) for Drowsiness Detection

Electroencephalography (EEG) is a method that measures the brain's electrical activity. As shown in Figure 3, it can be used to measure the heartbeat, eye blink and even major physical movement such as head movement. It can be used on humans or animals as subjects to get brain activity. It uses a special hardware that places sensors around the top of the head area to sense any electrical brain activity.

C. Drowsiness Detection using Face Detection System

Drowsiness can be detected by using face area detection. The methods to detect drowsiness within face area vary due to drowsiness signs are more visible and clearer to be detected at face area. From the face area, we can detect the eye's location. From eyes detection, we can say that there are four types of eyelid movement that can be used for drowsiness detection. They are completely open, completely close, and in the middle where the eyes are from open to close and vice versa. Figure 2 is an example of the image taken for detecting eyelid movement.

 The algorithm processes the images captured in a grey-scale method; where the colour from the images is then transformed into black and white. Working with black and white images is easier because only two parameters have to be measured. Edge detection is performed to detect the edges of eyes so that the value of eyelid area can be calculated. The problem occurring with this method is that the size of the eye might vary from one person to another. Someone may have small eyes and look like they are sleepy but some are not. Other than that, if the person is wearing glasses, there is an obstacle to detect eye regions. The images that are captured must be in a certain range from the camera because when the distance is far from  the camera, the images are blurred.

D. PERCLOS

Drowsiness can be captured by detecting the eye blinks and percentage of eye closure (PERCLOS). For eye blink detection, propose a method which learns the pattern of duration  of eyelid closed. This method measures the time for a person to close their eyes and if they are closed longer than the normal eye blink time, it is possible that the  person is falling asleep’.

The PERCLOS method proposes that drowsiness is measured by calculating the percentage of the  eyelid ‘drops’. Sets of eye open and eye closed have been stored in the software library to  be used as a parameter to differentiate whether the eyes are fully open or fully closed. For eyelids to  droop, it happens in much slower time as the person is slowly falling asleep. Hence, the  transition of the driver’s drowsiness can be recorded. Thus, PERCLOS method puts a proportional  value where when the eye is 80% closed, which it is nearly to fully close, it is assumed that the  driver is drowsy.

This method is not convenient to be used in real-time driving as it needs to fix the threshold value of  eye opening for the PERCLOS method to perform accurately. Both methods to detect drowsiness  using eye blink patterns and PERCLOS have the same problem where the camera needs to be  placed at a specific angle in order to get a good image of video with no disturbance of eyebrow  and shadow that cover the eyes.

E. Yawning Detection Method

Drowsiness of a person can be observed by looking at their face and behavior.  A method is proposed where drowsiness can be detected by mouth positioning and the  images were processed by using a cascade of classifiers that has been proposed by Viola-Jones for  faces. The images were compared with the set of images data for mouth and yawning. Some  people will close their mouth with their hand while yawning. It is an obstacle to get good images if a person is closing their mouth while yawning but yawning is definitely a sign of a person having drowsiness and fatigue. Fig 3 demonstrates the face of human when in Normal and Yawning condition

 Below are the Machine Learning methodologies that we studied from various research papers that are published on concurrent topics. Based on the classifiers used in these papers we have categorized them accordingly:

1. Convolutional Neural Network (CNN): A convolutional neural network (CNN or convnet) is a subset of machine learning. A CNN is a kind of network architecture for deep learning algorithms and is specifically used for image recognition and tasks that involve the processing of pixel data.

TABLE I
Review Based On CNN Algorithm

2. Support Vector Machine (SVM): Support Vector Machine or SVM is one of the most popular Supervised Learning algorithms, which is used for Classification as well as Regression problems. However, primarily, it is used for Classification problems in Machine Learning. The goal of the SVM algorithm is to create the best line or decision boundary that can segregate n-dimensional space into classes so that we can easily put the new data point in the correct category in the future. This best decision boundary is called a hyperplane.

TABLE II
Review Based On SVM Algorithm

3. Logistic Regression: Logistic regression is a supervised learning classification algorithm used to predict the probability of a target variable. The nature of target or dependent variable is dichotomous, which means there would be only two possible classes. In simple words, the dependent variable is binary in nature having data coded as either 1 (stands for success/yes) or 0 (stands for failure/no).

TABLE III
Review Based On Logistic Regression Algorithm

4. HAAR and OpenCV: Haar cascade is an algorithm that can detect objects in images, irrespective of their scale in image and location. The OpenCV library manages a repository containing all popular haar cascades that can be used for: Human face detection, Eye detection, Nose / Mouth detection, Vehicle detection

TABLE III
Review Based On HAAR and OPENCV

III. SYSTEM REQUIREMENT SPECIFICATION

A. Objectives

1. To detect the parameters that determine the level of drowsiness of the driver.
2. To analyse and predict the drowsy condition of the driver using ML techniques.
3. To investigate the physical changes of fatigue and drowsiness.
4. To develop a system that uses Eye Aspect Ratio (EAR) as a way to detect fatigue and drowsiness and alert the driver.

B. Project Scope

In this project, we have focused on the following procedures:

1. Basic concept of drowsiness detection system
2. Familiarize with the signs of drowsiness
3. Determine the drowsiness from these parameters

a. Eye blink

b. Area of the pupils detected at eyes

c. Yawning

4. Data collection and measurement.

5. Integration of the methods chosen.

6. Coding development and testing.

7. Complete testing and improvement.

C. System Requirements

1. Software Requirements Specification

a. Python 3 Libraries: Python is the basis of the program that we wrote. It utilizes many of the python libraries.

i NumPy: Prerequisite for Dlib.

ii SciPy: Used for calculating Euclidean distance between the eyelids.

iii. Playsound: Used for sounding the alarm.

iv. Dlib: This program is used to find the frontal human face and estimate its pose using face landmarks.

v. Imutils: Convenient functions written for Opencv.

vi. OpenCV: Used to get the video stream from the webcam.

b. Operating System- Windows or Ubuntu

2. Hardware Requirements Specification

a. Laptop with basic hardware: Used to run our code.

b. Webcam: Used to get the video feed.

D. System Implementation Plan

The framework is created utilizing the incremental model. The centre model of the framework is first created and afterwards augmented in this way in the wake of testing at each turn. The underlying undertaking of the skeleton was refined into expanding levels of ability. At the following incremental level, it might incorporate new execution backing and improvement.

Modular Division: The entire architecture is divided into 6 modules:

1. Face Detection: This module takes input from the camera and tries to detect a  face in the video input. The detection of the face is achieved through the Haar  classifiers mainly, the Frontal face cascade classifier. The face is detected in a rectangle format and converted to grayscale image and stored in the memory which  can be used for training the model.
2. Eye Detection: Since the model works on building a detection system for  drowsiness we need to focus on the eyes detect drowsiness. The eyes are  detected through the video input by implementing a haar classifier namely Haar Cascade Eye Classifier. The eyes are detected in rectangular formats.
3. Face Tracking: Due to the real-time nature of the project, we need to track the  faces continuously for any form of distraction. Hence the faces are continuously  detected during the entire time.
4. Eye Tracking: The input to this module is taken from the previous module. The eye's state is determined through Perclos algorithm.
5. Drowsiness Detection: In the previous module the frequency is calculated and if it remains 0 for a longer period then the driver is alerted for drowsiness through  an alert from the system.
6. Alert: The system will alert the driver if the drowsiness condition is detected by the use of a prediction system based on a machine learning approach.

IV. OTHER SPECIFICATIONS

A.  Limitations

1. Dependence on Ambient Light: With poor lighting conditions even though the face is easily detected, sometimes the system is unable to detect the eyes. So it gives an erroneous result which must be taken care of.
2. Optimum Range Required: When the distance between face and webcam is not at optimum range then certain problems are arising. Eyes are not detected with high accuracy which shows error in detection of drowsiness. The optimum distance range for drowsiness detection is set to 40-70 cm
3. Hardware Requirements: Our system was run on a PC with a configuration of 1.6GHz and 1GB RAM Pentium dual core processor. Though the system runs fine on higher configurations, when a system has an inferior configuration, the system may not be smooth and drowsiness detection will be slow. The problem was resolved by using dedicated hardware in real time applications, so there are no issues of frame buffering or slower detection.
4. Orientation of Face: When the face is tilted to a certain extent it can be detected, but beyond this our system fails to detect the face. So when the face is not detected, eyes are also not detected. This problem is resolved by using tracking functions which track any movement and rotation of the objects in an image. A trained classifier for tilted face and tilted eyes can also be used to avoid this kind of problem.
5. Poor Detection with Spectacles: When the driver wears glasses the system fails to detect eyes which is the most significant drawback of our system. This issue has not yet been resolved and is a challenge for almost all eye detection systems designed so far.
6. Problem with Multiple Faces: If more than one face is detected by the webcam, then our system gives an erroneous result.

B. Applications

1. Public transportation systems and private transport where accidents occur due to driver fatigue.
2. Vigilance on Security guards.
3. Highly secured Nuclear Power projects where continuous monitoring is necessary
4. Monitoring of soldiers in highly protected Military areas.
5. Monitoring of students in classrooms.

C. Future Work

Features of zoom in and zoom out can be added to the existing system. Future work may be to robotically zoom in on the eyes as soon as they are localized. This would reduce the vast subject of view in order to come across the eyes and a narrow view in order to detect fatigue. It will be too late to give the warning at that point. New methods can be discovered to generate warnings by analyzing different eye motion patterns. Use of 3D images of the face would increase the accuracy of localizing areas of detection. Warning methods can be improvised by giving SOS message alerts to the emergency contacts and by vibrating the driver's seat. The model can be improvised by use of night vision infrared cameras to detect facial features at night. Detection can be further enhanced by usage of cameras capable of detecting eyes in low light.

V. ACKNOWLEDGMENT

The satisfaction and euphoria that accompany the successful completion of any task would be impossible without the mention of the people who made it possible, whose constant guidance and encouragement crowned our efforts with success. I take this opportunity to express my profound gratitude to Dr. D. S. Bormane, Principal, AISSMS College of Engineering, for his constant support and encouragement. I would also like to thank Dr. S. V. Athawale, Head, Department of Computer Engineering, for his constant support and guidance. I express my gratitude to Mr. S. G.  Dhengre, Professor, project guide, for constantly monitoring the development of the project and setting up precise deadlines. His valuable suggestions were the motivating factors in completing the work. Finally, a note of thanks to the teaching and non teaching staff of the Department of Computer Engineering, for their cooperation extended to me, and my friends, who helped me directly or indirectly in the course of the project work.

Conclusion

The existing system consists of various approaches which are based on behavioral, vehicular and physiological aspects. Any of these approaches doesn\'t give 100% results. Every technique has some limitations which don\'t allow them to give perfect results. Thus on the basis of our study we conclude that if we try with a combination of two or more approaches such that one can reduce the limitations of another approach and thus help us in providing the best result. This can lead us in making a non intrusive and most efficient driver drowsiness detection system. We can combine some image processing approaches with some vehicular measures and physiological measures. Heart rate and respiration rates can be a good example of physiological measures which are clear indicators of drowsiness. To remove the intrusive nature of physiological measures we can use wireless sensors which can be effectively fitted in seat belts, seat covers etc.

References

[1] Driver Drowsiness Detection using Machine Learning with Visual Behaviour Shivanand Phulari1, Pranita Dalvi2, Pooja Girme3, Nikita Bhosale4, Payal Deshmukh5 , Mohini Pansare6 International Journal for Research in Applied Science & Engineering Technology (IJRASET) [2] Real-time Driver Drowsiness Detection for Android Application Using Deep Neural Networks Techniques , Rateb Jabbara* , Khalifa Al-Khalifa,b, Mohamed Kharbechea, Wael Alhajyaseena,Mohsen Jafaric, Shan Jiangc, The 9th International Conference on Ambient Systems, Networks, and Technologies (ANT 2018) [3] Real-Time Driver Drowsiness Detection using Computer Vision, Mahek Jain, Bhavya Bhagerathi, Sowmyarani C N,International Journal of Engineering and Advanced Technology (IJEAT) [4] Driver Fatigue Detection System, E. Rogado, J.L. García, R. Barea, L.M. Bergasa, Member IEEE and E. López,International Conference on Robotics and Biomimetics Bangkok, Thailand [5] A Real-time Driving Drowsiness Detection Algorithm With Individual Differences Consideration, FENG YOU 1 , XIAOLONG LI 1, YUNBO GONG 1, HAIWEI WANG 2, AND HONGYI LI 3,4, IEEE Access [6] Driver Drowsiness Detection Using Machine Learning, Prof. Swati Gade1, Kshitija Kamble2, Aishwarya Sheth3, Sakshi Patil4, Siddhi Potdar5, international Journal for Research in Applied Science & Engineering Technology (IJRASET) [7] Real Time Driver Drowsiness Detection Using a Logistic-Regression-Based Machine Learning Algorithm, Mohsen Babaeian, Nitish Bhardwaj, Bianca Esquivel, and Mohammad Mozumdar, 2016 IEEE Green Energy and Systems Conference (IGSEC) [8] Driver Fatigue Detection System, Yogesh Chellappa, Narendra Nath Joshi, and Vaishnavi Bharadwaj, IEEE International Conference Signal and Image Processing [9] Machine Learning and Gradient Statistics Based Real-Time Driver Drowsiness Detection, Cyun-Yi Lin*, Paul Chang#, Alan Wang# , and Chih-Peng Fan*, 2018 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW) [10] A Study on Feature Extraction Methods Used to Estimate a Driver’s Level of Drowsiness, Kyong He Lee, Whui Kim, Hyun Kyun Choi, Byung Tae Jang, International Conference on Advanced Communication Technology (ICACT) [11] Real-Time Driver Drowsiness Detection System Using Eye Aspect Ratio and Eye Closure Ratio, Sukrit Mehta, Sharad Dadhich, Sahil Gumber, Arpita Jadhav Bhatt, 2019 SUSCOM. Hosted by Elsevier SSRN. [12] A Multimodal Biometric System for Personal Identification Based on Deep Learning Approaches, Alaa S. Al-Waisy, Rami Qahwaji, Stanley Ipson, Shumoos Al-FahdawiSeventh International Conference on Emerging Security Technologies 2018 (EST) [13] Early Detection of Driver Drowsiness Utilizing Machine Learning based on Physiological Signals, Behavioral Measures, and Driving Performance,Jongseong Gwak1, Motoki Shino2, and Akinari Hirao, 2018 21st International Conference on Intelligent Transportation Systems (ITSC) [14] Detecting Drowsy Drivers Using Machine Learning Algorithms Majid Aksari,University of Wisconsin-Madison

Copyright

Copyright © 2022 Mr. S. G. Dhengre, Aishwarya Patil, Bhaskar Soman, Onkar Mirajkar, Rushikesh Suryawanshi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.